Automated three-dimensional image registration for longitudinal photoacoustic imaging.

Significance: Photoacoustic tomography (PAT) has great potential in monitoring disease progression and treatment response in breast cancer. However, due to variations in breast repositioning, there is a chance of geometric misalignment between images. Further, poor repositioning can affect light fluence distribution and imaging field-of-view, making images different from one another. The net effect is that it becomes challenging to distinguish between image changes due to repositioning effects and those due to true biological variations. Aim: The aim is to develop a three-dimensional image registration framework for geometrically aligning repeated PAT volumetric images, which are potentially affected by repositioning effects such as misalignment, changed radiant exposure conditions, and different fields-of-view. Approach: The proposed framework involves the use of a coordinate-based neural network to represent the displacement field between pairs of PAT volumetric images. A loss function based on normalized cross correlation and Frangi vesselness feature extraction at multiple scales was implemented. We refer to our image registration framework as MUVINN-reg, which stands for multiscale vesselness-based image registration using neural networks. The approach was tested on a longitudinal dataset of healthy volunteer breast PAT images acquired with the hybrid photoacoustic-ultrasound Photoacoustic Mammoscope 3 imaging system. The registration performance was also tested under unfavorable repositioning conditions such as intentional mispositioning, and variation in breast-supporting cup size between measurements. Results: A total of 13 pairs of repeated PAT scans were included in this study. MUVINN-reg showed excellent performance in co-registering each pair of images. The proposed framework was shown to be robust to image intensity shifts and field-of-view changes. Furthermore, MUVINN-reg could align vessels at imaging depths greater than 4 cm. Conclusions: The proposed framework will enable the use of PAT for quantitative and reproducible monitoring of disease progression and treatment response.

Just-in-time deep learning for real-time X-ray computed tomography.

Real-time X-ray tomography pipelines, such as implemented by RECAST3D, compute and visualize tomographic reconstructions in milliseconds, and enable the observation of dynamic experiments in synchrotron beamlines and laboratory scanners. For extending real-time reconstruction by image processing and analysis components, Deep Neural Networks (DNNs) are a promising technology, due to their strong performance and much faster run-times compared to conventional algorithms. DNNs may prevent experiment repetition by simplifying real-time steering and optimization of the ongoing experiment. The main challenge of integrating DNNs into real-time tomography pipelines, however, is that they need to learn their task from representative data before the start of the experiment. In scientific environments, such training data may not exist, and other uncertain and variable factors, such as the set-up configuration, reconstruction parameters, or user interaction, cannot easily be anticipated beforehand, either. To overcome these problems, we developed just-in-time learning, an online DNN training strategy that takes advantage of the spatio-temporal continuity of consecutive reconstructions in the tomographic pipeline. This allows training and deploying comparatively small DNNs during the experiment. We provide software implementations, and study the feasibility and challenges of the approach by training the self-supervised Noise2Inverse denoising task with X-ray data replayed from real-world dynamic experiments.

2DeteCT - A large 2D expandable, trainable, experimental Computed Tomography dataset for machine learning.

Recent research in computational imaging largely focuses on developing machine learning (ML) techniques for image reconstruction, which requires large-scale training datasets consisting of measurement data and ground-truth images. However, suitable experimental datasets for X-ray Computed Tomography (CT) are scarce, and methods are often developed and evaluated only on simulated data. We fill this gap by providing the community with a versatile, open 2D fan-beam CT dataset suitable for developing ML techniques for a range of image reconstruction tasks. To acquire it, we designed a sophisticated, semi-automatic scan procedure that utilizes a highly-flexible laboratory X-ray CT setup. A diverse mix of samples with high natural variability in shape and density was scanned slice-by-slice (5,000 slices in total) with high angular and spatial resolution and three different beam characteristics: A high-fidelity, a low-dose and a beam-hardening-inflicted mode. In addition, 750 out-of-distribution slices were scanned with sample and beam variations to accommodate robustness and segmentation tasks. We provide raw projection data, reference reconstructions and segmentations based on an open-source data processing pipeline.

Mitigating the Limited View Problem in Photoacoustic Tomography for a Planar Detection Geometry by Regularized Iterative Reconstruction.

The use of a planar detection geometry in photoacoustic tomography results in the so- called limited-view problem due to the finite extent of the acoustic detection aperture. When images are reconstructed using one-step reconstruction algorithms, image quality is compromised by the presence of streaking artefacts, reduced contrast, image distortion and reduced signal-to-noise ratio. To mitigate this, model-based iterative reconstruction approaches based on least squares minimisation with and without total variation regularization were evaluated using in-silico, experimental phantom, ex vivo and in vivo data. Compared to one-step reconstruction methods, it has been shown that iterative methods provide better image quality in terms of enhanced signal-to-artefact ratio, signal-to-noise ratio, amplitude accuracy and spatial fidelity. For the total variation approaches, the impact of the regularization parameter on image feature scale and amplitude distribution was evaluated. In addition, the extent to which the use of Bregman iterations can compensate for the systematic amplitude bias introduced by total variation was studied. This investigation is expected to inform the practical application of model-based iterative image reconstruction approaches for improving photoacoustic image quality when using finite aperture planar detection geometries.

Effects of awareness and task relevance on neurocomputational models of mismatch negativity generation.

Detection of regularities and their violations in sensory input is key to perception. Violations are indexed by an early EEG component called the mismatch negativity (MMN) - even if participants are distracted or unaware of the stimuli. On a mechanistic level, two dominant models have been suggested to contribute to the MMN: adaptation and prediction. Whether and how context conditions, such as awareness and task relevance, modulate the mechanisms of MMN generation is unknown. We conducted an EEG study disentangling influences of task relevance and awareness on the visual MMN. Then, we estimated different computational models for the generation of single-trial amplitudes in the MMN time window. Amplitudes were best explained by a prediction error model when stimuli were task-relevant but by an adaptation model when task-irrelevant and unaware. Thus, mismatch generation does not rely on one predominant mechanism but mechanisms vary with task relevance of stimuli.

3D deformable registration of longitudinal abdominopelvic CT images using unsupervised deep learning.

BACKGROUND AND OBJECTIVES: Deep learning is being increasingly used for deformable image registration and unsupervised approaches, in particular, have shown great potential. However, the registration of abdominopelvic Computed Tomography (CT) images remains challenging due to the larger displacements compared to those in brain or prostate Magnetic Resonance Imaging datasets that are typically considered as benchmarks. In this study, we investigate the use of the commonly used unsupervised deep learning framework VoxelMorph for the registration of a longitudinal abdominopelvic CT dataset acquired in patients with bone metastases from breast cancer. METHODS: As a pre-processing step, the abdominopelvic CT images were refined by automatically removing the CT table and all other extra-corporeal components. To improve the learning capabilities of the VoxelMorph framework when only a limited amount of training data is available, a novel incremental training strategy is proposed based on simulated deformations of consecutive CT images in the longitudinal dataset. This devised training strategy was compared against training on simulated deformations of a single CT volume. A widely used software toolbox for deformable image registration called NiftyReg was used as a benchmark. The evaluations were performed by calculating the Dice Similarity Coefficient (DSC) between manual vertebrae segmentations and the Structural Similarity Index (SSIM). RESULTS: The CT table removal procedure allowed both VoxelMorph and NiftyReg to achieve significantly better registration performance. In a 4-fold cross-validation scheme, the incremental training strategy resulted in better registration performance compared to training on a single volume, with a mean DSC of 0.929±0.037 and 0.883±0.033, and a mean SSIM of 0.984±0.009 and 0.969±0.007, respectively. Although our deformable image registration method did not outperform NiftyReg in terms of DSC (0.988±0.003) or SSIM (0.995±0.002), the registrations were approximately 300 times faster. CONCLUSIONS: This study showed the feasibility of deep learning based deformable registration of longitudinal abdominopelvic CT images via a novel incremental training strategy based on simulated deformations.

Comparison of convolutional neural network training strategies for cone-beam CT image segmentation.

BACKGROUND AND OBJECTIVE: Over the past decade, convolutional neural networks (CNNs) have revolutionized the field of medical image segmentation. Prompted by the developments in computational resources and the availability of large datasets, a wide variety of different two-dimensional (2D) and three-dimensional (3D) CNN training strategies have been proposed. However, a systematic comparison of the impact of these strategies on the image segmentation performance is still lacking. Therefore, this study aimed to compare eight different CNN training strategies, namely 2D (axial, sagittal and coronal slices), 2.5D (3 and 5 adjacent slices), majority voting, randomly oriented 2D cross-sections and 3D patches. METHODS: These eight strategies were used to train a U-Net and an MS-D network for the segmentation of simulated cone-beam computed tomography (CBCT) images comprising randomly-placed non-overlapping cylinders and experimental CBCT images of anthropomorphic phantom heads. The resulting segmentation performances were quantitatively compared by calculating Dice similarity coefficients. In addition, all segmented and gold standard experimental CBCT images were converted into virtual 3D models and compared using orientation-based surface comparisons. RESULTS: The CNN training strategy that generally resulted in the best performances on both simulated and experimental CBCT images was majority voting. When employing 2D training strategies, the segmentation performance can be optimized by training on image slices that are perpendicular to the predominant orientation of the anatomical structure of interest. Such spatial features should be taken into account when choosing or developing novel CNN training strategies for medical image segmentation. CONCLUSIONS: The results of this study will help clinicians and engineers to choose the most-suited CNN training strategy for CBCT image segmentation.

Efficient high cone-angle artifact reduction in circular cone-beam CT using deep learning with geometry-aware dimension reduction.

High cone-angle artifacts (HCAAs) appear frequently in circular cone-beam computed tomography (CBCT) images and can heavily affect diagnosis and treatment planning. To reduce HCAAs in CBCT scans, we propose a novel deep learning approach that reduces the three-dimensional (3D) nature of HCAAs to two-dimensional (2D) problems in an efficient way. Specifically, we exploit the relationship between HCAAs and the rotational scanning geometry by training a convolutional neural network (CNN) using image slices that were radially sampled from CBCT scans. We evaluated this novel approach using a dataset of input CBCT scans affected by HCAAs and high-quality artifact-free target CBCT scans. Two different CNN architectures were employed, namely U-Net and a mixed-scale dense CNN (MS-D Net). The artifact reduction performance of the proposed approach was compared to that of a Cartesian slice-based artifact reduction deep learning approach in which a CNN was trained to remove the HCAAs from Cartesian slices. In addition, all processed CBCT scans were segmented to investigate the impact of HCAAs reduction on the quality of CBCT image segmentation. We demonstrate that the proposed deep learning approach with geometry-aware dimension reduction greatly reduces HCAAs in CBCT scans and outperforms the Cartesian slice-based deep learning approach. Moreover, the proposed artifact reduction approach markedly improves the accuracy of the subsequent segmentation task compared to the Cartesian slice-based workflow.

Emulation of X-ray Light-Field Cameras.

X-ray plenoptic cameras acquire multi-view X-ray transmission images in a single exposure (light-field). Their development is challenging: designs have appeared only recently, and they are still affected by important limitations. Concurrently, the lack of available real X-ray light-field data hinders dedicated algorithmic development. Here, we present a physical emulation setup for rapidly exploring the parameter space of both existing and conceptual camera designs. This will assist and accelerate the design of X-ray plenoptic imaging solutions, and provide a tool for generating unlimited real X-ray plenoptic data. We also demonstrate that X-ray light-fields allow for reconstructing sharp spatial structures in three-dimensions (3D) from single-shot data.

Explorative Imaging and Its Implementation at the FleX-ray Laboratory.

In tomographic imaging, the traditional process consists of an expert and an operator collecting data, the expert working on the reconstructed slices and drawing conclusions. The quality of reconstructions depends heavily on the quality of the collected data, except that, in the traditional process of imaging, the expert has very little influence over the acquisition parameters, experimental plan or the collected data. It is often the case that the expert has to draw limited conclusions from the reconstructions, or adapt a research question to data available. This method of imaging is static and sequential, and limits the potential of tomography as a research tool. In this paper, we propose a more dynamic process of imaging where experiments are tailored around a sample or the research question; intermediate reconstructions and analysis are available almost instantaneously, and expert has input at any stage of the process (including during acquisition) to improve acquisition or image reconstruction. Through various applications of 2D, 3D and dynamic 3D imaging at the FleX-ray Laboratory, we present the unexpected journey of exploration a research question undergoes, and the surprising benefits it yields.

A cone-beam X-ray computed tomography data collection designed for machine learning.

Unlike previous works, this open data collection consists of X-ray cone-beam (CB) computed tomography (CT) datasets specifically designed for machine learning applications and high cone-angle artefact reduction. Forty-two walnuts were scanned with a laboratory X-ray set-up to provide not only data from a single object but from a class of objects with natural variability. For each walnut, CB projections on three different source orbits were acquired to provide CB data with different cone angles as well as being able to compute artefact-free, high-quality ground truth images from the combined data that can be used for supervised learning. We provide the complete image reconstruction pipeline: raw projection data, a description of the scanning geometry, pre-processing and reconstruction scripts using open software, and the reconstructed volumes. Due to this, the dataset can not only be used for high cone-angle artefact reduction but also for algorithm development and evaluation for other tasks, such as image reconstruction from limited or sparse-angle (low-dose) scanning, super resolution, or segmentation.

Single-pixel camera photoacoustic tomography.

Since it was first demonstrated more than a decade ago, the single-pixel camera concept has been used in numerous applications in which it is necessary or advantageous to reduce the channel count, cost, or data volume. Here, three-dimensional (3-D), compressed-sensing photoacoustic tomography (PAT) is demonstrated experimentally using a single-pixel camera. A large area collimated laser beam is reflected from a planar Fabry­Pérot ultrasound sensor onto a digital micromirror device, which patterns the light using a scrambled Hadamard basis before it is collected into a single photodetector. In this way, inner products of the Hadamard patterns and the distribution of thickness changes of the FP sensor­induced by the photoacoustic waves­are recorded. The initial distribution of acoustic pressure giving rise to those photoacoustic waves is recovered directly from the measured signals using an accelerated proximal gradient-type algorithm to solve a model-based minimization with total variation regularization. Using this approach, it is shown that 3-D PAT of imaging phantoms can be obtained with compression rates as low as 10%. Compressed sensing approaches to photoacoustic imaging, such as this, have the potential to reduce the data acquisition time as well as the volume of data it is necessary to acquire, both of which are becoming increasingly important in the drive for faster imaging systems giving higher resolution images with larger fields of view.

Improved EEG source localization with Bayesian uncertainty modelling of unknown skull conductivity.

Electroencephalography (EEG) source imaging is an ill-posed inverse problem that requires accurate conductivity modelling of the head tissues, especially the skull. Unfortunately, the conductivity values are difficult to determine in vivo. In this paper, we show that the exact knowledge of the skull conductivity is not always necessary when the Bayesian approximation error (BAE) approach is exploited. In BAE, we first postulate a probability distribution for the skull conductivity that describes our (lack of) knowledge on its value, and model the effects of this uncertainty on EEG recordings with the help of an additive error term in the observation model. Before the Bayesian inference, the likelihood is marginalized over this error term. Thus, in the inversion we estimate only our primary unknown, the source distribution. We quantified the improvements in the source localization when the proposed Bayesian modelling was used in the presence of different skull conductivity errors and levels of measurement noise. Based on the results, BAE was able to improve the source localization accuracy, particularly when the unknown (true) skull conductivity was much lower than the expected standard conductivity value. The source locations that gained the highest improvements were shallow and originally exhibited the largest localization errors. In our case study, the benefits of BAE became negligible when the signal-to-noise ratio dropped to 20 dB.

Real-time cardiovascular MR with spatio-temporal artifact suppression using deep learning-proof of concept in congenital heart disease.

PURPOSE: Real-time assessment of ventricular volumes requires high acceleration factors. Residual convolutional neural networks (CNN) have shown potential for removing artifacts caused by data undersampling. In this study, we investigated the ability of CNNs to reconstruct highly accelerated radial real-time data in patients with congenital heart disease (CHD). METHODS: A 3D (2D plus time) CNN architecture was developed and trained using synthetic training data created from previously acquired breath hold cine images from 250 CHD patients. The trained CNN was then used to reconstruct actual real-time, tiny golden angle (tGA) radial SSFP data (13 × undersampled) acquired in 10 new patients with CHD. The same real-time data was also reconstructed with compressed sensing (CS) to compare image quality and reconstruction time. Ventricular volume measurements made using both the CNN and CS reconstructed images were compared to reference standard breath hold data. RESULTS: It was feasible to train a CNN to remove artifact from highly undersampled radial real-time data. The overall reconstruction time with the CNN (including creation of aliased images) was shown to be >5 × faster than the CS reconstruction. In addition, the image quality and accuracy of biventricular volumes measured from the CNN reconstructed images were superior to the CS reconstructions. CONCLUSION: This article has demonstrated the potential for the use of a CNN for reconstruction of real-time radial data within the clinical setting. Clinical measures of ventricular volumes using real-time data with CNN reconstruction are not statistically significantly different from gold-standard, cardiac-gated, breath-hold techniques.

Three dimensional photoacoustic tomography in Bayesian framework.

The image reconstruction problem (or inverse problem) in photoacoustic tomography is to resolve the initial pressure distribution from detected ultrasound waves generated within an object due to an illumination by a short light pulse. Recently, a Bayesian approach to photoacoustic image reconstruction with uncertainty quantification was proposed and studied with two dimensional numerical simulations. In this paper, the approach is extended to three spatial dimensions and, in addition to numerical simulations, experimental data are considered. The solution of the inverse problem is obtained by computing point estimates, i.e., maximum a posteriori estimate and posterior covariance. These are computed iteratively in a matrix-free form using a biconjugate gradient stabilized method utilizing the adjoint of the acoustic forward operator. The results show that the Bayesian approach can produce accurate estimates of the initial pressure distribution in realistic measurement geometries and that the reliability of these estimates can be assessed.

Equivalent-Source Acoustic Holography for Projecting Measured Ultrasound Fields Through Complex Media.

Holographic projections of experimental ultrasound measurements generally use the angular spectrum method or Rayleigh integral, where the measured data are imposed as a Dirichlet boundary condition. In contrast, full-wave models, which can account for more complex wave behavior, often use interior mass or velocity sources to introduce acoustic energy into the simulation. Here, a method to generate an equivalent interior source that reproduces the measurement data is proposed based on gradient-based optimization. The equivalent-source can then be used with full-wave models (for example, the open-source k-Wave toolbox) to compute holographic projections through complex media including nonlinearity and heterogeneous material properties. Numerical and experimental results using both time-domain and continuous-wave sources are used to demonstrate the accuracy of the approach.

Model-Based Learning for Accelerated, Limited-View 3-D Photoacoustic Tomography.

Recent advances in deep learning for tomographic reconstructions have shown great potential to create accurate and high quality images with a considerable speed up. In this paper, we present a deep neural network that is specifically designed to provide high resolution 3-D images from restricted photoacoustic measurements. The network is designed to represent an iterative scheme and incorporates gradient information of the data fit to compensate for limited view artifacts. Due to the high complexity of the photoacoustic forward operator, we separate training and computation of the gradient information. A suitable prior for the desired image structures is learned as part of the training. The resulting network is trained and tested on a set of segmented vessels from lung computed tomography scans and then applied to in-vivo photoacoustic measurement data.

Accelerated high-resolution photoacoustic tomography via compressed sensing.

Current 3D photoacoustic tomography (PAT) systems offer either high image quality or high frame rates but are not able to deliver high spatial and temporal resolution simultaneously, which limits their ability to image dynamic processes in living tissue (4D PAT). A particular example is the planar Fabry-Pérot (FP) photoacoustic scanner, which yields high-resolution 3D images but takes several minutes to sequentially map the incident photoacoustic field on the 2D sensor plane, point-by-point. However, as the spatio-temporal complexity of many absorbing tissue structures is rather low, the data recorded in such a conventional, regularly sampled fashion is often highly redundant. We demonstrate that combining model-based, variational image reconstruction methods using spatial sparsity constraints with the development of novel PAT acquisition systems capable of sub-sampling the acoustic wave field can dramatically increase the acquisition speed while maintaining a good spatial resolution: first, we describe and model two general spatial sub-sampling schemes. Then, we discuss how to implement them using the FP interferometer and demonstrate the potential of these novel compressed sensing PAT devices through simulated data from a realistic numerical phantom and through measured data from a dynamic experimental phantom as well as from in vivo experiments. Our results show that images with good spatial resolution and contrast can be obtained from highly sub-sampled PAT data if variational image reconstruction techniques that describe the tissues structures with suitable sparsity-constraints are used. In particular, we examine the use of total variation (TV) regularization enhanced by Bregman iterations. These novel reconstruction strategies offer new opportunities to dramatically increase the acquisition speed of photoacoustic scanners that employ point-by-point sequential scanning as well as reducing the channel count of parallelized schemes that use detector arrays.

Electrical Stimulation of the Human Cerebral Cortex by Extracranial Muscle Activity: Effect Quantification With Intracranial EEG and FEM Simulations.

OBJECTIVE: Electric fields (EF) of approx. 0.2 V/m have been shown to be sufficiently strong to both modulate neuronal activity in the cerebral cortex and have measurable effects on cognitive performance. We hypothesized that the EF caused by the electrical activity of extracranial muscles during natural chewing may reach similar strength in the cerebral cortex and hence might act as an endogenous modality of brain stimulation. Here, we present first steps toward validating this hypothesis. METHODS: Using a realistic volume conductor head model of an epilepsy patient having undergone intracranial electrode placement and utilizing simultaneous intracranial and extracranial electrical recordings during chewing, we derive predictions about the chewing-related cortical EF strength to be expected in healthy individuals. RESULTS: We find that in the region of the temporal poles, the expected EF strength may reach amplitudes in the order of 0.1-1 V/m. CONCLUSION: The cortical EF caused by natural chewing could be large enough to modulate ongoing neural activity in the cerebral cortex and influence cognitive performance. SIGNIFICANCE: The present study lends first support for the assumption that extracranial muscle activity might represent an endogenous source of electrical brain stimulation. This offers a new potential explanation for the puzzling effects of gum chewing on cognition, which have been repeatedly reported in the literature.

Simulating transcranial direct current stimulation with a detailed anisotropic human head model.

Transcranial direct current stimulation (tDCS) is a noninvasive brain stimulation technique able to induce long-lasting changes in cortical excitability that can benefit cognitive functioning and clinical treatment. In order to both better understand the mechanisms behind tDCS and possibly improve the technique, finite element models are used to simulate tDCS of the human brain. With the detailed anisotropic head model presented in this study, we provide accurate predictions of tDCS in the human brain for six of the practically most-used setups in clinical and cognitive research, targeting the primary motor cortex, dorsolateral prefrontal cortex, inferior frontal gyrus, occipital cortex, and cerebellum. We present the resulting electric field strengths in the complete brain and introduce new methods to evaluate the effectivity in the target area specifically, where we have analyzed both the strength and direction of the field. For all cerebral targets studied, the currently accepted configurations produced sub-optimal field strengths. The configuration for cerebellum stimulation produced relatively high field strengths in its target area, but it needs higher input currents than cerebral stimulation does. This study suggests that improvements in the effects of transcranial direct current stimulation are achievable.